Apache Iceberg: Streaming Options

TL;DR

• Apache Flink with the native Iceberg connector is the gold standard for true streaming and CDC into Iceberg with exactly-once semantics.
• Spark Structured Streaming offers low-friction, micro-batch ingestion to Iceberg for near real-time analytics at enterprise scale.
• Flink CDC pipelines (Debezium, JDBC CDC) to Iceberg give the cleanest path from OLTP to governed analytical tables.
• Apache Beam can target Iceberg via Flink/Spark runners or custom sinks on GCP-centric estates, but requires stricter commit discipline.
• Hive-to-Iceberg streaming is a transitional option for legacy estates; it reduces risk but carries latency and operability trade-offs.

Abstract & Executive Observations

Enterprises are moving streaming data from “raw files on object storage” into governed, queryable tables that uphold ACID guarantees, time travel, and schema evolution. Apache Iceberg has emerged as a de facto table format for that destination layer across AWS, Azure, and GCP. The question is not whether to target Iceberg, but how to get streaming data into it with operational rigour and predictable cost.

Our thesis: organisations should standardise on five streaming patterns to Iceberg, in priority order: (1) Flink + Iceberg, (2) Spark Structured Streaming + Iceberg, (3) Flink CDC + Iceberg, (4) Beam + Iceberg, and (5) Hive/Tez + Iceberg for legacy convergence. These options are not equivalents; they offer distinct latency, semantics, and governance characteristics that matter in production. The gap we routinely see is not the absence of tools, but the absence of disciplined table management: compaction, manifest maintenance, partition evolution, and snapshot retention. Without this, teams trade “streaming” for a future small-file storm and rising query costs.

Iceberg’s atomic commits, snapshot isolation, and partition evolution fit streaming precisely because each write produces a clean, auditable snapshot. With the right engine integrations—Flink and Spark foremost—you get transactional writes and exactly-once semantics. Paired with a robust catalog (Hive Metastore, AWS Glue, Nessie, or the Iceberg REST Catalog), and monitored with Prometheus/Grafana, you can hit sub-minute freshness without sacrificing reliability. This article ranks the five viable options, explains how to implement them, and shows the security and performance trade-offs we recommend clients consider up front.

The Current Architectural Challenge

Streaming to object stores was once an expedient compromise; today, it’s a liability at scale. Legacy approaches—custom Spark micro-batches writing raw Parquet, ad hoc consumers dumping append-only files, or home-grown upsert logic—generate three chronic issues:

• Small files and query sprawl: High-frequency, low-volume writes create thousands of tiny data files and an explosion of manifests. Query engines spend more time on planning and metadata scanning than on reading data, driving up both latency and cost.
• Consistency blind spots: Without atomic commits, consumers might read partially written datasets, see duplicates, or miss late-arriving records. The result is correctness risk, production “mystery bugs,” and brittle downstream models.
• Schema and partition ossification: Fixed partitioning aligned to early assumptions (for example, by day) breaks under evolving workloads (hourly freshness, session-based partitions, or region-based access). Schema drift either breaks jobs or is ignored until a costly refactor.

We see these patterns across sectors—from retail telemetry to banking transactions to public-sector log pipelines. Teams often bolt on “quick” fixes—compaction jobs that run nightly, regex-based schema inference, or heavier Spark clusters to mask small-file penalties—only to discover rising cost curves and operational instability. The hidden costs include:

• Manual compaction debt: Jobs that rewrite files run too slowly, miss windows, or cause read contention.
• Scaling pain: Increasing ingest rates or new upstream sources force rework of partitioning and bucketing strategies.
• Unenforced governance: Lineage, audit trails, and data retention policies become inconsistent across stovepiped pipelines.

Apache Iceberg is designed to address these faults. By committing writes atomically and materialising each write as a snapshot, Iceberg enables time travel, isolation, and repeatable reasoning about data’s state at any point. It further supports partition evolution, hidden partitioning (transform-based), and metadata for incremental planning. The missing link for many enterprises is choosing the right streaming engine integration and implementing disciplined table management—commit cadence, file sizing, compaction/optimise commands, and sensible snapshot retention—so that the benefits hold under continuous load.

Technical Visualization: Why legacy streaming-to-files breaks under scale

Figure 1: Legacy streaming jobs write raw files directly, causing non-atomic views, small-file storms, and metadata staleness—leading to slow, brittle queries and hard-to-govern datasets.

Implementation & Orchestration Strategy

You have five viable options for streaming into Iceberg. We rank them by operational maturity, semantic guarantees, and ecosystem support.

• Option 1 — Apache Flink + Iceberg (best overall): True streaming with exactly-once semantics, native upserts/deletes, and incremental queries. Works across Kafka, Kinesis, Pub/Sub, Pulsar, and CDC sources.
• Option 2 — Spark Structured Streaming + Iceberg: Micro-batch semantics with strong production tooling and broad skills availability. Ideal for near real-time (tens of seconds to minutes).
• Option 3 — Flink CDC (Debezium/JDBC) + Iceberg: The cleanest path for capturing change from OLTP to Iceberg, preserving keys and ordering for upsert/merge-on-read or rewrite-on-write.
• Option 4 — Apache Beam + Iceberg: Good for GCP-heavy architectures via Flink/Spark runners or custom Iceberg sinks. Requires care with commit semantics and file sizing.
• Option 5 — Hive/Tez + Iceberg (transitional): Allows legacy estates to stream into Iceberg with fewer rewrites. Accept higher latency and less flexible semantics while converging architecture.

Below we detail when to choose each option and how to implement them safely.

• Option 1 — Apache Flink + Iceberg

When to choose

You need sub-second to low-second latency, exactly-once semantics, and support for upserts/deletes. You plan to run continuous jobs with high throughput.

Architecture

Flink sources (Kafka, Kinesis via connectors, Pub/Sub via Beam I/O or community connectors, Pulsar I/O, Debezium CDC) → Flink Table/SQL or DataStream API → Iceberg Sink (Flink Iceberg connector) → Catalog (Hive/Glue/REST/Nessie) → Object store.

Semantics

With Flink’s checkpointing and two-phase commit, writes are atomic per checkpoint. Iceberg snapshots provide isolation for readers. Upserts and deletes can be modelled via primary-key tables or position deletes.

Operational guidance

• Set checkpoint intervals to 5–60 seconds; tune by throughput. The commit cadence should balance staleness vs small files.

  • Target file sizes of 128–512 MB to reduce planning overhead; Flink’s file compaction or Iceberg’s optimise actions can consolidate small files.
  • Use hidden partition transforms (for example, day(ts), hour(ts)) and evolve partitions as usage patterns change.
  • Enable incremental planning to reduce read cost for downstream queries.
  • Monitor commit duration and checkpoint lags through Prometheus/Grafana; alert on rising small-file counts and manifest growth.

• Option 2 — Spark Structured Streaming + Iceberg

When to choose

You have strong Spark skills; near real-time (micro-batch) is acceptable; you want a predictable, simpler operational model.

Architecture

Sources (Kafka, Kinesis via connectors, Pub/Sub via Dataflow or spark-connector bridges) → Spark Structured Streaming → Iceberg DataSource → Catalog (Glue/HMS/REST/Nessie) → Object store.

Semantics

Exactly-once within micro-batch boundaries when idempotency is enforced (checkpointing, deduplication). Writes are append or merge-based. Each micro-batch creates a consistent snapshot.

Operational guidance

• Use micro-batch intervals between 10–60 seconds. Avoid too small intervals that cause small files and snapshot bloat.

  • Leverage DataFrame writes with options that control target file size. Batch compaction/optimise periodically, or use Auto-Optimize (if available in your stack) carefully.
  • For upserts, use merge-into on primary-keyed tables or write deletes via Iceberg’s row/position deletes; ensure streaming job has deterministic key handling.

• Option 3 — Flink CDC (Debezium/JDBC) + Iceberg

When to choose

Your primary use case is database change capture (MySQL, Postgres, Oracle, SQL Server), and you need referentially consistent upserts/deletes into Iceberg.

Architecture

Debezium/CDC source → Flink CDC connectors → Flink SQL/Table API → Iceberg Sink → Catalog. Ideal for bronze-to-silver layering with CDC semantics preserved.

Semantics

Exactly-once across checkpoints; proper handling of inserts/updates/deletes into Iceberg keyed tables. Supports replay/time travel for audits.

Operational guidance

• Configure parallelism to respect table key cardinality and maintain ordering where necessary.

  • Ensure WAL/binlog retention in the source database to cover recovery windows.
  • Validate backfill processes; use Flink’s bounded source mode for historical loads and then switch to streaming mode.

• Option 4 — Apache Beam + Iceberg

When to choose

GCP-centric organisations or hybrid estates that already standardise on Beam/Dataflow. You’re comfortable with either running Beam on Flink/Spark runners for the Iceberg sink or maintaining a custom Iceberg sink with write-ahead/commit coordination.

Architecture

Beam pipeline → Flink/Spark runner (preferred for robust Iceberg sink support) → Iceberg Writer → Catalog → Object store. Alternatively, write intermediate files and ingest via Iceberg actions, though that adds latency and complexity.

Semantics

Achievable exactly-once depends on runner and sink implementation. Production-grade patterns use a two-phase commit sink and Beam’s checkpointing signals via the runner.

Operational guidance

• Prefer runners that support robust checkpoint/commit semantics (Flink runner). Align windowing and triggers with file sizing and commit cadence.

  • Add compaction as a managed step or schedule Iceberg optimise on the target tables.

• Option 5 — Hive/Tez + Iceberg (transitional)

When to choose

You have extensive Hive streaming or Tez-based ingestion and need an incremental path to Iceberg with minimal re-platforming.

Architecture

Hive streaming job or Tez pipeline → Iceberg tables via Iceberg-Hive integration → HMS/Glue catalog → Object store. Best used as an interim state while planning Flink or Spark migration.

Semantics

Typically higher-latency, more batch-like behaviours. Works for 5–60 minute SLAs but not ideal for sub-minute streaming.

Operational guidance

• Plan for compaction/optimise jobs to correct small-file patterns from frequent, small Hive tasks.

  • Treat this as a stepping stone; align table layout and partitioning with your eventual streaming posture so that migration is non-disruptive.

Beyond selection, the most common success factor is disciplined table management. For all options:

• Catalogs and governance:
• Choose a resilient catalog: AWS Glue for AWS-centric estates, Hive Metastore for on-prem or hybrid, Nessie/REST Catalog for Git-like branching and CI/CD patterns.
• Standardise on Iceberg snapshot retention policies; keep enough history for audits (for example, 7–30 days) without overloading manifests.

• Use branches/tags for write-audit-publish (WAP) in higher-risk changes and schema migrations.

• Table layout and partitioning:

• Prefer hidden partition transforms (for example, hour(ts), bucket(user_id, 16), truncate(country, 3)); avoid direct path-based partitioning coupled to column values.

• Embrace partition evolution: as query patterns change (for example, hourly reporting to near real-time), evolve from day(ts) to hour(ts) without rewriting history immediately.

• Compaction and maintenance:

• Schedule optimise operations after peak ingest windows or continuously with throttling. Prioritise:

  • Rewrite small data files into larger ones.
  • Rewrite manifests to keep planning time predictable.
  • Expire snapshots and clean orphan files to control storage costs.

• Monitoring and SLOs:

• Track: commit duration, checkpoint lag (Flink), micro-batch run time (Spark), small-file counts, average data file size, manifests per table, snapshot count, and read planning time from Trino/Spark.
• Export metrics to Prometheus; set Grafana dashboards with SLO-aligned thresholds (for example, commit < 5 s, file size median > 128 MB).

Architecture Blueprint: Ranked streaming options into Iceberg

Figure 2: The five ranked options feeding Iceberg. Flink (Options 1 and 3) leads for true streaming and CDC. Spark serves near real-time with micro-batches. Beam fits GCP-centric estates. Hive/Tez bridges legacy.

Implementation Detail: Exactly-once streaming commit to Iceberg

Figure 3: Flink’s checkpoint barriers coordinate a two-phase commit to Iceberg. Data and delete files are flushed, manifests prepared, and the catalog atomically publishes a new snapshot. Query engines read the consistent snapshot N.

Security, Performance, and Trade-offs

Security and governance

• Access control: Use catalog-backed permissions (Glue, HMS with Ranger/Apache Sentry alternatives, or REST Catalog with IAM) to enforce table- and namespace-level access. Pair with object-store policies to prevent out-of-band writes.

• Encryption: At-rest encryption via S3 SSE-KMS, GCS CMEK, or ADLS keys. For especially sensitive domains, consider client-side encryption before write, acknowledging the impact on query pushdown.

• Auditability: Iceberg snapshots are an audit trail. Use tags/branches for WAP and regulated releases; persist commit metadata (user, job id, git sha) for compliance.
• Row/column constraints: Implement row filters and column masking in the query layer (Trino/Spark) and back them with catalog policies; Iceberg’s metadata can support incremental planning for allowed partitions.

Performance realities

• Small files: The primary enemy of performance. Tune writer target sizes and commit intervals. Use optimise (rewrite data files) and rewrite manifests regularly.

• Commit contention: Very high commit frequency will grow snapshots and manifest churn; scale cadence to throughput. In multi-writer scenarios, coordinate commit times or leverage partition/distribution strategies to minimise conflicts.

• Upserts and deletes cost more: Expect higher write amplification due to delete files and merge semantics. Budget more resources and schedule compactions to convert position deletes into re-written data files where feasible.

• Query engines: Trino excels with Iceberg’s metadata for planning; Spark SQL offers deep integration and complex transformations. Choose engines per workload; mix and match is normal.

• Operational trade-offs by option:

• Flink + Iceberg: Highest semantic quality; requires deeper ops maturity (checkpointing, state backends, savepoints). Outstanding for continuous, high-rate streams and CDC.
• Spark + Iceberg: Simpler to adopt for Spark estates; lower ops cost but micro-batch latency and small-file risks need careful handling.
• Flink CDC + Iceberg: Best for OLTP-derived analytics; requires careful source DB retention and backfill strategies.
• Beam + Iceberg: Flexibility on GCP/hybrid; invest in a robust sink implementation and commit discipline.
• Hive/Tez + Iceberg: Lower change budget; higher latency and more batch-like behaviour; plan for migration.

Parting Thoughts on Industry Direction

Over the next 12–24 months, we expect three shifts. First, Iceberg’s REST Catalog and branch-aware workflows (including Project Nessie) will move from innovation to standard operating practice, enabling CI/CD for data—feature branches, WAP, and promotion to production snapshots. Second, streaming engines will increasingly normalise upserts/deletes as first-class patterns. Flink already leads; Spark is closing the gap with improved streaming merge support. Third, compaction will become continuous and adaptive: optimise actions triggered by table-level SLOs, not cron.

For leaders, the winning strategy is to standardise on a small set of patterns—Flink and Spark foremost—backed by disciplined table management. If you would like to benchmark ingest latency, cost per TB, or compaction overheads across engines and clouds, we can share anonymised metrics and reference architectures. We’ve implemented these options in banking, health, and public sector; the right choice depends on your latency targets, skills, and governance posture. Let’s discuss what “good” looks like for your estate.

Recommended Reading

• A practical guide to Iceberg table layout and partition evolution for streaming workloads (/insights/ai/related-topic-1/)
• Exactly-once in practice: Flink, checkpoints, and transactional sinks demystified (/insights/ai/related-topic-2/)
• Governing data products on open table formats: access controls, WAP, and lineage (/insights/ai/related-topic-3/)

About the Research Team

The MNB Technology Research Team consists of senior architects and engineers with backgrounds spanning global banking, healthcare, retail, defence, and public sector. Our work focuses on enterprise AI, secure distributed systems, and resilient operational delivery.

Work with Us

Ready to make · new · better?

Contact us to discuss how this architecture applies to your specific environment.